{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "41bc133f-3bf6-4541-a19e-1be1ddf05cf2",
   "metadata": {},
   "source": [
    "# 导包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cd5f4065-0fa0-44aa-975f-5421bde6afa6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7b297ae-e953-4ef1-a87a-60389698d33a",
   "metadata": {},
   "source": [
    "# 加载数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "95d8a8d0-9951-4637-b012-996c56377a51",
   "metadata": {},
   "outputs": [],
   "source": [
    "def deal_text(x):\n",
    "    new_li = []\n",
    "    for i in x.split():\n",
    "        new_li.append(int(i))\n",
    "    return new_li\n",
    "    \n",
    "def deal_data(path):\n",
    "    df = pd.read_csv(path,sep='\\t')\n",
    "    df['text'] = df['text'].apply(lambda x:deal_text(x))\n",
    "    df['len_text'] = df['text'].apply(lambda x:len(x))\n",
    "    df['max_idx_word'] = df['text'].apply(lambda x:max(x))\n",
    "    df['min_idx_word'] = df['text'].apply(lambda x:min(x))\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a127bde9-0a9c-46d8-9a66-be0bd707d30e",
   "metadata": {},
   "outputs": [],
   "source": [
    "path1 = './datas/train_set.csv'\n",
    "path2 = './datas/test_a.csv'\n",
    "df1 = deal_data(path1)\n",
    "df2 = deal_data(path2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4d9385a0-e62d-4e7c-9116-f18e0d572b35",
   "metadata": {},
   "outputs": [],
   "source": [
    "df1.to_csv('./datas/pre_train.cav',index=False)\n",
    "df2.to_csv('./datas/pre_test.cav',index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abff29b6-c329-4f02-90b6-27f8f6b0d0a6",
   "metadata": {},
   "source": [
    "- 词索引范围(0~7549)\n",
    "- 序列长度范围 (2~57921)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "546f2fc4-acbd-41e8-925f-b03d38c4f5ba",
   "metadata": {},
   "source": [
    "- 损失值突然飙升是典型的梯度爆炸现象\n",
    "- 可能原因：\n",
    "    1. 学习率过高（当前0.001）\n",
    "    2. 缺少梯度裁剪\n",
    "    3. 网络层过深（双向LSTM堆叠）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cecf0826-225d-4231-9873-df5709906096",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
